As mobile computing technology becomes more ever-present in our daily lives, mobile device users become more and more reliant on functionality provided by their mobile devices. Ideally, mobile devices, or other computing devices, should allow users to specify complex behaviors. Allowing users to specify complex behaviors gives rise to customization of device actions with respect to different situations or conditions to better serve the user's needs.
An online article called “Automate your Motorola Razr with Smart Actions” by Jack Wallen, published Jan. 17, 2012 on TechRepublic.Com (see URL www.techrepublic.com/blog/smartphones/automate-your-motorola-razr-with-smart-actions/4215) describes the state of a configurable smart phone. The application on the device allows users to specify a number of complex behaviors by creating rules which may utilize the many basic functions of the device. For example, a user can create a rule that causes the device to be silenced based on its knowing a particular location (say home) and a particular time of day (say 6:00-7:00 pm). This application supports only touch-base controls and does not support technologies capable of making even simple rule creation totally effortless and intuitive such as through speech recognition or natural language processing. The technology cannot convert spoken input signals to create rules governing complex behaviors for the device to perform.
Others have put forth an effort toward offering user control of device behavior. One example includes U.S. patent application publication 2011/0254792 to Waters et al. titled “User Interface to Provide Enhanced Control of an Application Program”, filed as an international application on Dec. 18, 2009. Waters merely focuses on providing support for touch-based controls rather than a natural multi-modal dialog interaction. Still another example includes U.S. Pat. No. 7,154,862 to Krzyzanowski titled “Device Control System, Method, and Apparatus for Server-Based or Peer-to-Peer Network Environments”, filed Feb. 23, 2004. Krzyzanowski seeks to provide a unified interface via handheld devices. Still further, U.S. Pat. No. 7,302,394 to Baray et al. titled “Front-End Device Independence for Natural Interaction Platform”, filed Dec. 20, 2002, contemplates providing a natural language interpretation system that provides commands to back end applications.
U.S. Pat. No. 8,346,563 B1 to David Hjelm, Robert Kriiger, Bjorn Giilsdorff, Ebba Birgitta Gustavii, and Maria Cheadle titled “System and methods for delivering advanced natural language interaction applications”, patent date: Jan. 1, 2013 describes a system for delivering advanced natural language interaction applications, comprising a dialog interface module, a natural language interaction engine, a solution data repository component operating comprising at least one domain model, at least one language model, and a plurality of flow elements and rules for managing interactions with users, and an interface software module. Users submit requests via a network which are preprocessed by the dialog interface module and subsequently transmitted to the natural language interaction engine. Using language recognition rules stored in the solution data repository and the determined semantic meaning or user intent, the natural language interaction engine forms and delivers an appropriate response or makes an appropriate action based on the request. Hjelm et al. describe a generic approach to human-computer conversational dialog management. The work fails to distinguish a system responding in the sense of “executing an action” from system responding that entails complex device behavior. Hjelm et al. fails to address the creation or use of triggers in the creation of complex device behaviors. Additionally, the work does not discuss the creation of complex device behaviors from one or more function primitives.
U.S. Pat. No. 7,751,884 B2 to David Ternes, Dong M. Birkholz, David W. Yost, and James A. Esler titled “Flexible Neural Stimulation Engine” issued Jul. 6, 2010 regards an implantable medical pacemaker device that monitors device state and the previous behavior of the device relative to state. Input to the device affects device state which, in turn, can cause the device to take action based upon the change in device state and neural event timers. Ternes et al. makes some progress regarding the monitoring of device state and taking an action as a consequence of the state of the device. Ternes et al.; however, fails to distinguish a reactive action from a complex behavior. Ternes et al. do not disclose to any method for the creation of and use of triggers in the creation and performance of complex device behaviors constructed from function primitives. Additionally, the work fails to describe any application of the methods to additional problem domains.
U.S. Patent Publication Number US 2011/0161076 A1 to Bruce L. Davis, Tony F. Rodriguez, William Y. Conwell and Geoffrey B. Rhoads titled “Intuitive Computing Methods and Systems” published Jun. 30, 2011 describes a smart phone configured to sense audio, imagery or other stimulus data of its user's environment. Based upon such input, the system can act autonomously in accordance with or fulfillment of the inferred desires of the user. Davis et al. make progress in methods to determine user intent and enabling autonomous reaction to user input. Additionally, Davis et al. describe progress in the use of phone sensors to determine device state and subsequently tailor device behaviors or trigger an action. Davis et al. fails to address however reactions that constitutes complex device behaviors. They further fail to describe the creation and use of triggers in the creation and performance of complex device behaviors constructed from function primitives.
U.S. Patent Publication Number US 2009/0112832 A1 to Eser Kandogan, Jie Lu and Michelle Xue Zhou titled “Intelligent Content Assistance” published Apr. 30, 2009 describes a method for generating one or more context-sensitive content recommendations by detecting the information needs of a user, retrieving one or more content-recommendation templates that match the detected information needs and instantiating the retrieved templates with one or more parameter values to generate recommended contents. The system generates these recommendations during business processes. It dynamically monitors and detects the needs of a user in a particular context. Additionally, it retrieves content-recommendation templates that appear to match the information needs of the user, instantiating them using information from user input, context, interaction history, system-learned query and content models and external sources. Context, interaction history and system-learned models are updated dynamically for learning and adaptation. Kandogan et al. makes some progress toward using environmental, situational or contextual factors in the detection of the information needs of a user. The work further advances the retrieval of content recommendation templates that appear to match the information needs of the user. Kandogan et al. additionally makes progress in regards to the use of context, interaction history, system learned models and content models domains in the determination of system responses. Kandogan et al. however lacks insight with respect to any complex device behavior. Kandogan et al. further fail to describe the creation and use of triggers in the creation and performance of complex device behaviors constructed from function primitives.
U.S. Pat. No. 8,332,224 B2 to Philippe Di Cristo, Chris Weider and Robert A. Kennewick titled “System and Method of Supporting Adaptive Misrecognition Conversational Speech” issued Dec. 11, 2012 describes a system and methods that support multimodal input including natural language speech input. The system interprets spoken utterances or other received non-spoken inputs using a conversational interface that includes a conversational speech analyzer, a general cognitive model, an environmental model, and a personalized cognitive model to determine context, domain knowledge, and invoke prior information. The system does not address sensor data or the functional states of a device. The system creates, stores and uses extensive personal profile information for each user, thereby improving the reliability of determining the context of the speech or non-speech communication and presenting the expected results for a particular question or command. Cristo et al. exhibit progress regarding the use of a personalized cognitive model in determining the context of speech and non-speech communication. The Cristo et al. system does not address sensor data or the functional states of a device however. Cristo et al. also fails to address the conversion of signals to complex device behavior. Cristo et al. also fails to address the creation and use of triggers in the creation and performance of complex device behaviors constructed from function primitives.
U.S. Patent Publication Number US 2013/0031476 A1 to Emmett Coin, Deborah Dahl and Richard Mandelbaum titled “Voice Activated Virtual Assistant” published Jan. 31, 2013 describes a multimodal conversational interface that interprets spoken natural language utterances, visual cues, images and/or other forms of input. The system includes a conversation manager that relies on a set of functions defining very flexible adaptive scripts. During interaction with a user, the conversation manager obtains information from the user, refining or defining more accurately what information is required by the user, asking the user for additional information to determine the user's intent. User responses result in the selection of scripts or subscripts relevant to subsequent actions. The system may obtain additional data relevant to the dialog from local sensors or remotely from other sources during an interaction. Remote sources are accessed by activation of an appropriate function such as a search engine. Coin et al. make progress regarding the interactive determination of user intent, the selection of script or subscripts that are relevant to subsequent behaviors and the use of sensor data. Coin et al. however fail to provide insight into the creation of complex device behaviors. Coin et al. however fail to provide insight into the creation of complex device behaviors. Coin et al. also fails to address the creation and use of triggers in the creation and performance of complex device behaviors constructed from function primitives.
These references fail to provide insight into how users can be enabled to construct complex device behaviors based on natural interactions with a device environment, especially in a context where previous interactions can affect future device states. Furthermore, the references fail to provide any insight into using primitives to instantiate a future device behavior. There thus remains a need for systems and methods that address both the conversion of signals to complex device behavior and creating and using triggers for complex device behavior from function primitives.
These and all other extrinsic materials discussed herein are incorporated by reference in their entirety. Where a definition or use of a term in an incorporated reference is inconsistent or contrary to the definition of that term provided herein, the definition of that term provided herein applies and the definition of that term in the reference does not apply.
Unless the context dictates the contrary, all ranges set forth herein should be interpreted as being inclusive of their endpoints, and open-ended ranges should be interpreted to include commercially practical values. Similarly, all lists of values should be considered as inclusive of intermediate values unless the context indicates the contrary.
Thus, with the difficulty of using graphical interfaces, particularly on small devices with limited display area, the increase need for effective assistant behaviors on computing devices, and the greater power and flexibility of spoken interfaces, there is still a need for a spoken method for user creation of complex behaviors.